Run-time bottleneck detection

ABSTRACT

A method of detecting a bottleneck in a data flow program includes executing the data flow program, wherein the data flow program comprises a kernel, determining a percentage of time the kernel spends on a computation, and determining the kernel to be a bottleneck upon comparing the percentage of time the kernel spends on the computation to a condition.

BACKGROUND

1. Technical Field

The present application relates to bottleneck detection, and moreparticularly to the detection of a bottleneck at run-time.

2. Discussion of Related Art

Analyzing and identifying performance bottlenecks in computer systemscan be difficult, requiring labor intensive data generation, expertknowledge of the application, libraries, middleware, operating systemand hardware, and analytical tools.

Existing proposals for detecting central processor unit (CPU) andinput/output (I/O) bottlenecks in parallel directed acyclic graph (DAG)based data flow programs assume global knowledge of data flow graphtopology. Such knowledge is not always readily available.

BRIEF SUMMARY

According to an exemplary embodiment of the present disclosure, a methodof detecting a bottleneck in a data flow program includes executing, bya processor, computer readable instructions constituting the data flowprogram, wherein the data flow program comprises a kernel, determining apercentage of time the kernel spends on a computation, and determiningthe kernel to be a bottleneck upon comparing the percentage of time thekernel spends on the computation to a condition.

According to an exemplary embodiment of the present disclosure, a methodof detecting a bottleneck in a data flow program includes executing, bya processor, computer readable instructions constituting the data flowprogram, wherein the data flow program comprises a kernel, measuring anexecution delay of the kernel, receiving at least one delay message froma neighboring kernel in the data flow program, and determining thekernel to be the bottleneck upon comparing the execution delay of thekernel to a delay indicated by the delay message.

According to an exemplary embodiment of the present disclosure, a methodof detecting a bottleneck in a data flow program includes executing, bya processor, computer readable instructions constituting the data flowprogram, wherein the data flow program comprises a plurality of kernels,determining a percentage of time that each of the plurality of kernelspends on a respective computation, comparing the percentages to acondition, and determining at least one of the plurality of kernels tobe a bottleneck upon determining that the condition is satisfied.

According to an exemplary embodiment of the present disclosure, a methodof detecting a bottleneck in a data flow program includes executing, bya processor, computer readable instructions constituting the data flowprogram, wherein the data flow program comprises a plurality of kernels,measuring an execution delay of each of the plurality of kernels, andpropagating a delay message in the data flow program, wherein the delaymessage indicates a highest execution delay of the plurality of kernelsand a certain kernel of the plurality of kernels corresponding to thehighest execution delay.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Preferred embodiments of the present disclosure will be described belowin more detail, with reference to the accompanying drawings:

FIGS. 1A-C show flow diagrams illustrating exemplary conditionsaccording to embodiments of the present disclosure;

FIG. 2 is a graph about an exemplary data flow graph upon whichembodiment of the present disclosure may be applied.

FIGS. 3A-B are graphs according to an embodiment of the presentdisclosure; and show two exemplary delay channels that can be used inexemplary communicating bottleneck detection methods.

FIG. 4 is a flowchart that depicts a general execution of a kernel in adata flow graph before any exemplary bottleneck detection algorithm isapplied.

FIG. 5 is a flowchart that depicts an execution of a kernel when thefirst exemplary communicating bottleneck detection is applied.

FIG. 6 is a flowchart that depicts the details of the first exemplarycommunicating bottleneck detection method within a kernel.

FIG. 7 is a flowchart that depicts an execution of a kernel with anexemplary local bottleneck detection method.

FIGS. 8A-B are state transition diagrams for describing the processes ofan initiating kernel and all other kernels in the second exemplarycommunicating bottleneck detection method; and

FIG. 9 is an exemplary system for execution a method according to anembodiment of the present disclosure.

DETAILED DESCRIPTION

According to an embodiment of the present disclosure, a bottleneck maybe detected for data flow programs in an on-line context.

A bottleneck is computation task, or kernel, whose delay affects thethroughput of the data flow program. The data flow program may beexecuting on a computer system such as a personal computer, inmulti-processor type computer systems, and in the context ofmassively-parallel data processing on shared-nothing architecture,including cloud architecture.

It should be understood that a bottleneck may occur in other contextsand that the systems and methods described herein are not limited to theexamples discussed.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limiting of the present disclosure. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the presentdisclosure. The terminology used herein was chosen to best explain theprinciples of embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand embodiments disclosedherein.

According to an exemplary embodiment of the present disclosure, dataflow programs of interest may be modeled as a directed graph 200 asshown in FIG. 2, where a vertex, e.g., 201, represents a computingkernel and an edge, e.g., 202, represents a communication channelthrough which data are passed from one kernel to another.

According to an embodiment of the present disclosure, an edge (u,v) is acommunication channel between two vertices. The edge is unidirectionaland may be modeled as a First-In-First-Out (FIFO) queue. The interactionbetween kernels u and v follows a producer-consumer pattern over a fixedsized queue. A producer, u, which attempts to write to the full queue isneeded to block until at least one entry in the queue becomes available.Likewise, a consumer, v, which attempts to read from the queue isrequired to block until at least one data item is written to the queueby producer u.

According to an embodiment of the present disclosure, one firing of akernel may be defined as one completion of a kernel including reading ofinput data from the incoming queues and writing of output data tooutgoing queues. For a firing of a kernel, a certain number of datatokens are needed for each incoming queue. The number of data tokens iseither known prior to runtime or at the time of a firing.

According to an embodiment of the present disclosure, the execution timeof a firing of a kernel may change at runtime.

A kernel in a data flow program may execute a code segment in accordancewith FIG. 4. Assuming that a termination signal has not been received(401), the kernel's execution is blocked until each of its input datachannels provides sufficient number of input data and each of its outputdata channels has enough space to accommodate the computation results(402). Once all of the input data arrives, the kernel reads the data andremoves them from the input channels, executes its computation task, andwrites the results to its output channels (403).

Exemplary bottleneck detection methods can identify bottlenecks in adata flow program in various cases. For example, a bottleneck may bedetected when there is no central agent that has the knowledge of acomplete topology of the data flow graph. Stated generally, each kernelcan work individually. Further, exemplary methods may be applied whenkernels are distributed across connected computing platforms. Exemplarymethods may detect bottlenecks when cycles exist in the data flowprogram.

Various bottleneck-detection methods are applicable to data flowprograms satisfying various assumptions.

Exemplary bottleneck-detection methods may be used on the data flowprogram with blocking execution semantics. Blocking execution semanticsblock a kernel's execution according to input/output availability of thekernel. More specifically, a kernel is required to wait for input datato consume; also, a kernel is required to wait for an available storagespace for its output; the kernel executes only if input data areavailable and space for the storing output is sufficient. Otherwise, thekernel waits until those conditions are satisfied. In the presentdisclosure, an action of the kernel is a computation of the kernelexcluding waiting for input data and outputting data to storage.

Assuming the above blocking execution semantics, a bottleneck-detectionmethod may use one or more conditions to determine whether a kernel is abottleneck. Exemplary conditions are shown FIG. 1A-C. Block 101 of FIG.1A shows a delay time comparison, wherein the bottleneck kernel'sexecution delay of one firing is the largest among all of the kernels inthe data flow program. Block 102 of FIG. 1B shows that the bottleneckkernel's execution may not be blocked due to the absence of its inputdata. Block 103 of FIG. 1C shows that the bottleneck kernel's executionmay not be blocked due to the absence of storage space for storing aresult.

Bottleneck-detection methods may be executed by each kernel in a dataflow program. Further, each exemplary bottleneck-detection methodindividually measures the execution delay of its host kernel orindividually identifies a condition(s) on its input/output queues. Inone data flow program all kernels use the same bottleneck-detectionmethod; different exemplary bottleneck-detection methods may not bemixed in the same data flow program.

Communicating Bottleneck Detection: Two exemplary bottleneck-detectionmethods are described based on execution delay as shown in block 101 ofFIG. 1A, and in each of these two methods the information on kerneldelays may be communicated among kernels. These exemplarybottleneck-detection methods may be called communicatingbottleneck-detection methods.

A first exemplary implementation of a communicating bottleneck-detectionmethod is detailed in FIGS. 5 and 6, while the state transition diagramsfor a second implementation are given in FIGS. 8A-B. For communicatingbottleneck-detection methods, no central agent is needed to determine alargest among all kernels' delays. It may be sufficient for each kernelto be aware of only its immediate upstream and downstream kernels, i.e.,the immediate neighboring nodes in the graph. For a sink kernel such askernel 6 in FIG. 2, if any, it is required to know the source kernel,for example, kernel 1. A source kernel of a dataflow program is a kernelthat has no upstream kernels but only downstream kernels as itsneighboring nodes. A dataflow program may have one or more sourcekernel. In case of the existence of a plurality of source kernels, apseudo source kernel may be introduced in the context of directedacyclic graph. The pseudo source kernel may have all the source kernelsas downstream neighbors. A kernel running an exemplarybottleneck-detection method may select a largest delay observed so far,including delay measurements received from its upstream neighbor(s) andof its own delay, and outputs the largest delay observed by the kernelto its downstream receiver(s). It should be understood that thecondition may be variously configured (e.g., largest, 2 largest delays,delays greater than a threshold, etc.).

It should be understood a data flow program may include one or morekernels. Herein the terms “node” and “kernel” may be used to refer tosoftware components, while “processing node” may be used to indicate acomponent implemented in hardware.

Construction of Delay Channels: The communicating methods use at leastone channel between kernels for communicating delay messages in additionto a data communication channel. These additional channels are delaychannels.

The construction of the delay channels is orthogonal to thecommunicating bottleneck-detection method and can be constructedoff-line. Off-line methods for constructing the delay channels may bebased on the topology of a given data flow graph.

According to the first example in FIG. 3A, the first method forconstructing delay channels may be applied to acyclic data flow graphs.Since the given data flow graph is an acyclic data flow graph, it may beassumed that there is a single source kernel s and a sink kernel t. Theconstruction of the delay channels closely follows the data channels inthe flow graph. For each data channel (i, j), a parallel delay channelfrom kernel i to j is added. In addition a delay channel may be addedconnecting the sink kernel t to the source kernel s.

The second exemplary method for augmenting a data flow program appliesto both cyclic and acyclic data flow graphs and is shown in FIG. 3B. Toconstruct the delay channels for a data flow graph G={V,E}, G's spanningtree T={V,B} is determined. Notice that the set of tree edges B is asubset of the graph edges E. Without loss of generality, assume that Thas one root vertex r and a non-empty set of leaf vertices L⊂V. For eachtree edge (i, j)εB, a delay channel is built from kernel i to j.Further, for each leaf kernel l, a delay channel (l,r) is builtconnecting l to the root kernel r. Notice that a channel from the leafto the root does not correspond to any tree edge in B, and may not havea corresponding edge in E either.

Delay channels constructed by any of the exemplary methods, can be usedby any communicating bottleneck-detection method.

First Exemplary Communicating Method: For the communicatingbottleneck-detection type methods, each kernel in a data flow programperforms its main computation and a bottleneck-detection method. FIG. 5shows a general structure of the kernel code in which the exemplarycommunicating bottleneck detection method is adopted. Each kernel isaugmented with additional input and output delay channels. The kernelmeasures the elapsed execution time of the work time in FIG. 5.

More particularly, a bottleneck status is initialized to false (501).The kernel's execution is blocked until each of its input data channelsprovides sufficient input data and each of its output data channels hasenough space to accommodate the computation results (503). A start timeand an end time are measured or determined (504) and the work time ofthe kernel is determined as the end time minus the start time (505). Akernel is determined to be a bottleneck at block 506 (for example, seeFIG. 1A). A kernel's execution repeats until receipt of a terminationsignal at block 502.

At block 506, a kernel's bottleneck detection method may use itsmeasured work time and incoming delay messages to decide whether thekernel is a bottleneck. The outgoing delay message is calculated in thebottleneck detection method and sent to the outgoing delay channels. Adelay message is short and simple. Its exact format may depend on thebottleneck-detection algorithm. An exemplary bottleneck detection methodmay attempt to update the maximum effective delay of the data flowprogram as shown in FIG. 6.

In FIG. 6, the maximum of the incoming delays is determined (601). Thecurrent measured work time of the kernel may be normalized by the numberof firings of the kernel. If the maximum incoming delay is sufficientlysmaller then the normalized current work time (602) the kernel isdetermined to be a bottleneck (603). Otherwise the kernel is determinednot to be a bottleneck (604). In the case of a bottleneck (603), thelarger of the maximum delay and the current work time is set to anoutgoing delay channel (605).

Various methods can be used to compare the quantities of delays. Forexample, in one comparison, a method may determine if the maximumincoming delay (num1) is sufficiently smaller than the current measuredwork time (num2) (602). In this case the following formula may be used:diff=num2−num1;return(diff≧num2*DIFF THRESHOLD)?true:false

Second Exemplary Communicating Method: The second exemplarycommunicating bottleneck detection method adopts more explicit statetransitions. More specifically, kernels repeatedly iterate throughmultiple states, e.g., three states measure, compare, and decide, whilethe data flow program is running. The transition from one state toanother is triggered by incoming and outgoing delay messages. It may beassumed there is only one source kernel s in the data flow program.

In the first state measure a kernel measures the delay of each firing ofits main computation task, but does not communicate. The delaymeasurements can be averaged across multiple firings. The source kernelinitiates the transition from state measure to the second state compare.The state transition at the source kernel may be triggered by certainpredetermined criteria, e.g., if the measured system throughput degradessignificantly.

An example three-state diagram for a source node is shown in FIG. 8A.Block 801 and the condition 802 show the first measuring state and thecondition for initiating the state to change to second state compare(803). Counterparts for a non-source kernel are shown in FIG. 8B.

In the second state compare (803 in FIG. 8A) the source kernel sends outits own measured delay to each of the downstream receivers of thekernel's output delay channels. After receiving a delay message from anyof its input delay channels, a non-source kernel enters state compare.In state compare the kernel then compares the delay it has received sofar with its own measured delay, and saves the maximum (809 in FIG. 8B).After the kernel receives one delay message from each of its input delaychannels, it sends out the maximum delay to the downstream receivers ofits output delay channels (810 in FIG. 8B).

After sending out the maximum delay message a kernel enters state decide(805 in FIG. 8A and 811 in FIG. 8B). In this state the source kernelwaits in non-blocking fashion until all of its upstream neighbors havesent in one delay message with a delay measurement. It then selects themaximum delay among all the received delay measurements. If the maximumdelay equals its own delay, it declares itself as a bottleneck. Thismaximum delay is then sent to each of its output delay channels, and thesource kernel returns to state measure. Each of the non-source kernels,after receiving a second delay message, compares the delay measurementin the delay message with its own delay. If the received delay is thesame as the kernel's own measured delay, the kernel declares itself as abottleneck. The kernel also forwards the received delay to each of itsoutput delay channels. After receiving delay from each of its inputchannels, a kernel returns to state measure.

Local (Non-Communicating) Exemplary Bottleneck Detection: According toan embodiment of the present disclosure, a local bottleneck-detectionmethod determines whether a kernel is a bottleneck according to how muchtime the kernel spends on computation as compared to the amount timewhen the kernel is blocked. Unlike communicating bottleneck-detectionmethods, the local bottleneck-detection method does not require anadditional delay message or delay channel. FIG. 7 is a flow diagram ofexemplary kernel code for a local bottleneck-detection method.

Accessing the ratio of time when a kernel spends on computation to thetime when the kernel is blocked is used as an exemplary method fordetermining a bottleneck based on the conditions shown in FIGS. 1B and1C.

Referring to FIG. 7, a program start time is determined (701) and abottleneck status is initialized to false (501). An accumulated worktime W is initialized to zero (703). Blocks 502, 503 and 504 arediscussed above with reference to FIG. 5. Suppose an elapsed runtime ofthe program at the end of an iteration is T (708), while the kernelspends W time (707) doing useful work and (T−W) time being blocked. Abottleneck-detection method determines the ratio R=W/T (709). If R isgreater than a predetermined threshold (709), e.g., 0.95, the kernel isdeclared as a bottleneck (710) and not otherwise (711).

As described above, exemplary bottleneck detection algorithms may bedescribed as communicating and local methods. Each method may beexecuted by each kernel. The communicating methods use extra delaycommunicating channels. In contrast, the local method does not needdelay channels and may only measure the performance of its host kernel.

The methodologies of embodiments of the disclosure may be particularlywell-suited for use in an electronic device or alternative system.Accordingly, embodiments of the present disclosure may take the form ofan entirely hardware embodiment or an embodiment combining software andhardware aspects that may all generally be referred to herein as a“processor”, “circuit,” “module” or “system.” Furthermore, embodimentsof the present disclosure may take the form of a computer programproduct embodied in one or more computer readable medium(s) havingcomputer readable program code stored thereon.

Any combination of one or more computer usable or computer readablemedium(s) may be utilized. The computer-usable or computer-readablemedium may be a computer readable storage medium. A computer readablestorage medium may be, for example but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer-readablestorage medium would include the following: a portable computerdiskette, a hard disk, a random access memory (RAM), a read-only memory(ROM), an erasable programmable read-only memory (EPROM or Flashmemory), an optical fiber, a portable compact disc read-only memory(CD-ROM), an optical storage device, a magnetic storage device, or anysuitable combination of the foregoing. In the context of this document,a computer readable storage medium may be any tangible medium that cancontain or store a program for use by or in connection with aninstruction execution system, apparatus or device.

Computer program code for carrying out operations of embodiments of thepresent disclosure may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Embodiments of the present disclosure are described above with referenceto flowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products. It will be understood that eachblock of the flowchart illustrations and/or block diagrams, andcombinations of blocks in the flowchart illustrations and/or blockdiagrams, can be implemented by computer program instructions.

These computer program instructions may be stored in a computer-readablemedium that can direct a computer or other programmable data processingapparatus to function in a particular manner, such that the instructionsstored in the computer-readable medium produce an article of manufactureincluding instruction means which implement the function/act specifiedin the flowchart and/or block diagram block or blocks.

The computer program instructions may be stored in a computer readablemedium that can direct a computer, other programmable data processingapparatus, or other devices to function in a particular manner, suchthat the instructions stored in the computer readable medium produce anarticle of manufacture including instructions which implement thefunction/act specified in the flowchart and/or block diagram block orblocks.

For example, FIG. 9 is a block diagram depicting an exemplary system forbottleneck detection. The system 901 may include a processor 902, memory903 coupled to the processor (e.g., via a bus 904 or alternativeconnection means), as well as input/output (I/O) circuitry 905-906operative to interface with the processor 902. The processor 902 may beconfigured to perform one or more methodologies described in the presentdisclosure, illustrative embodiments of which are shown in the abovefigures and described herein.

It is to be appreciated that the term “processor” as used herein isintended to include any processing device, such as, for example, onethat includes a central processing unit (CPU) and/or other processingcircuitry (e.g., digital signal processor (DSP), microprocessor, etc.).Additionally, it is to be understood that the term “processor” may referto a multi-core processor that contains multiple processing cores in aprocessor or more than one processing device, and that various elementsassociated with a processing device may be shared by other processingdevices.

Both for a local bottleneck detection method and a communicationbottleneck detection method a multi-core processor or multi-processorsystem may be used in a single system 901, where each of the cores in amulti-core processor or each of the processor in the multi-processorsystem may be a processing node. Kernels in a dataflow program are runon multiple cores and/or processing devices in the single system. At thesame time, both methods may be used in a distributed system where atleast one additional system 907 is connected with the system 901 via anetwork 908. Kernels in a dataflow program may be run on multiple coresand/or processing devices in each single system within the distributedsystem.

The term “memory” as used herein is intended to include memory and othercomputer-readable media associated with a processor or CPU, such as, forexample, random access memory (RAM), read only memory (ROM), fixedstorage media (e.g., a hard drive), removable storage media (e.g., adiskette), flash memory, etc. Furthermore, the term “I/O circuitry” asused herein is intended to include, for example, one or more inputdevices (e.g., keyboard, mouse, etc.) for entering data to theprocessor, and/or one or more output devices (e.g., printer, monitor,etc.) for presenting the results associated with the processor.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

Although illustrative embodiments of the present disclosure have beendescribed herein with reference to the accompanying drawings, it is tobe understood that the disclosure is not limited to those preciseembodiments, and that various other changes and modifications may bemade therein by one skilled in the art without departing from the scopeof the appended claims.

What is claimed is:
 1. A method of detecting a bottleneck in a data flowprogram, the method comprising: executing, by a processor, computerreadable instructions constituting the data flow program, wherein thedata flow program comprises a kernel; identifying that a condition ispresent on an input/output queue of the kernel, wherein a condition onthe input queue of the kernel is a state in which the kernel's executionis blocked by a waiting for data to consume and a condition on theoutput queue of the kernel is a state in which the kernel's execution isblocked by a waiting for available space to store an output of thekernel; determining a time spent waiting for satisfaction of theidentified condition on the input/output queue; determining a percentageof time the kernel spends on a computation; and determining the kernelto be a bottleneck upon comparing the percentage of time the kernelspends on the computation to the time spent waiting for satisfaction ofthe identified condition.
 2. The method of claim 1, wherein determiningthe percentage of time the kernel spends on the computation comprises:determining a work time of the kernel as an accumulation of time spentby a processing node performing the computation of the kernel;determining a total time of the kernel as a total runtime of thedataflow program until a time of a measurement within the kernel; anddetermining the percentage of time the kernel spends on the computationby dividing the work time by the total time.
 3. The method of claim 2,wherein the work time excludes a time waiting for input data and a timeoutputting data to storage.
 4. The method of claim 2, wherein the pointof the measurement is a known set of instructions of the kernel.
 5. Themethod of claim 1, wherein the kernel is blocked due to lack of inputdata.
 6. The method of claim 1, wherein the kernel is blocked due tolack of space for a result of the computation.
 7. The method of claim 1,wherein comparing the percentage of time the kernel spends includescalculating a ratio of the time the kernel spends on the computationcompared to the time the kernel spends waiting for satisfaction of theidentified condition and determining whether the calculated ratio meetsor exceeds a predetermined threshold.
 8. The method of claim 1, furthercomprising a computer program product for detecting the bottleneck inthe data flow program, the computer program product comprising anon-transitory computer readable storage medium having computer readableprogram code embodied therewith for performing the method of claim
 1. 9.A method of detecting a bottleneck in a data flow program, the methodcomprising: executing, by a processor, non-transitory computer readableinstructions constituting the data flow program, wherein the data flowprogram comprises a first kernel; measuring an execution delay of thefirst kernel; receiving at least one delay message from an immediateupstream or immediate downstream kernel, the immediate upstream orimmediate downstream kernel being a kernel whose node on a graph of thedata flow program is directly connected to a node of the first kernel,either immediately before or immediately after the first kernel; anddetermining the first kernel to be the bottleneck upon comparing theexecution delay of the first kernel to a delay indicated by the delaymessage, wherein the first kernel is only aware of its immediateupstream and immediate downstream kernels which are immediateneighboring nodes on the graph.
 10. The method of claim 9, wherein eachkernel of the data flow program is associated with a respective dataflow node, and each kernel propagates a maximum delay in a respectivedelay message, wherein the maximum delay is determined from among theexecution delay of the first kernel and the delay indicated by the delaymessage from the immediately upstream or immediately downstream kernel.11. The method of claim 9, wherein measuring the execution delay of thefirst kernel comprises: determining a start time of the first kernel;determining an end time of the first kernel; and determining adifference between the end time and the start time.
 12. The method ofclaim 9, further comprising initiating a propagation of delay messagesfor a bottleneck detection.
 13. The method of claim 9, furthercomprising a computer program product for detecting the bottleneck inthe data flow program, the computer program product comprising anon-transitory computer readable storage medium having computer readableprogram code embodied therewith for performing the method of claim 9.14. A method of detecting a bottleneck in a data flow program, themethod comprising: executing, by a processor, non-transitory computerreadable instructions constituting the data flow program, wherein thedata flow program comprises a plurality of kernels; identifying that acondition is present on an input/output queue of each of the pluralityof kernels, wherein a condition on the input queue of the kernel is astate in which the kernel's execution is blocked by a waiting for datato consume and a condition on the output queue of the kernel is a statein which the kernel's execution is blocked by a waiting for availablespace to store an output of the kernel; determining a percentage of timethat each of the plurality of kernel spends on a respective computationrelative to a time that each of the plurality of the plurality ofkernels spends waiting for satisfaction of each of the identifiedconditions; and determining at least one of the plurality of kernels tobe a bottleneck upon determining that the determined percentage of timeexceeds a predetermined threshold.
 15. The method of claim 14, whereindetermining the percentage of time the each of the plurality of kernelspends on a respective computation comprises: determining a work time ofeach kernel as an accumulation of time spent by a processing nodeperform the computation of each kernel; determining a total time of eachkernel as a total runtime of the dataflow program until a time of ameasurement with the respective kernel; and determining the percentageof time each kernel spends on the computation by dividing the work timeby the total time of each respective kernel.
 16. The method of claim 15,wherein the point of the measurement is a known set of instructions ofthe respective kernel.
 17. The method of claim 14, wherein the kernel isblocked due to at least one of a lack of input data and a lack of spacefor a result of the computation.
 18. The method of claim 14, furthercomprising a computer program product for detecting the bottleneck inthe data flow program, the computer program product comprising anon-transitory computer readable storage medium having computer readableprogram code embodied therewith for performing the method of claim 14.19. A method of detecting a bottleneck in a data flow program, themethod comprising: executing, by a processor, non-transitory computerreadable instructions constituting the data flow program, wherein thedata flow program comprises a plurality of kernels; measuring anexecution delay of each of the plurality of kernels; and propagating adelay message in the data flow program, wherein the delay messageindicates a highest execution delay of the plurality of kernels and acertain kernel of the plurality of kernels corresponding to the highestexecution delay, and wherein propagating the delay message includespassing along a delay message from an immediate upstream kernel of theplurality of kernels to an immediate downstream kernel of the pluralityof kernels, the immediate upstream kernel being a kernel whose node on agraph of the data flow program is directly connected to a nodeimmediately before a node on the graph representing the immediatelydownstream kernel, wherein each kernel is only aware of its immediateupstream and immediate downstream kernels which are immediateneighboring nodes on the graph.
 20. The method of claim 19, wherein eachkernel of the data flow program is associated with a respective dataflow node, and each kernel propagates a maximum delay in the delaymessage, wherein the maximum delay is determined from among respectiveexecution delays of the kernel and the delay message received from aneighboring kernel.
 21. The method of claim 19, wherein measuring theexecution delay of the kernel comprises: determining a start time of thekernel; determining an end time of the kernel; and determining adifference between the end time and the start time.